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Background/Aims: Three-dimensional (3D) imaging is gaining popularity and has been partly adopted in laparoscopic surgery or ro- 
botic surgery but has not been applied to gastrointestinal endoscopy As a first step, we conducted an experiment to evaluate whether 
images obtained by conventional gastrointestinal endoscopy could be used to acquire quantitative 3D information. 

Methods: Two endoscopes (GIF-H260) were used in a Borrmann type I tumor model made of clay The endoscopes were calibrated by 
correcting the barrel distortion and perspective distortion. Obtained images were converted to gray-level image, and the characteristics of 
the images were obtained by edge detection. Finally, data on 3D parameters were measured by using epipolar geometry, two view geom- 
etry, and pinhole camera model. 

Results: The focal length if) of endoscope at 30 mm was 258.49 pixels. Two endoscopes were fixed at predetermined distance, 12 mm 
(dn). After matching and calculating disparity (v 2 -Vi), which was 106 pixels, the calculated length between the camera and object (L) was 
29.26 mm. The height of the object projected onto the image (h) was then applied to the pinhole camera model, and the result of H 
(height and width) was 38.21 mm and 41.72 mm, respectively. Measurements were conducted from 2 different locations. The measure- 
ment errors ranged from 2.98% to 7.00% with the current Borrmann type I tumor model. 

Conclusions: It was feasible to obtain parameters necessary for 3D analysis and to apply the data to epipolar geometry with conven- 
tional gastrointestinal endoscope to calculate the size of an object. 
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INTRODUCTION 

It is difficult to get a grasp of depth with one eye. Human 
eyes perceive objects using two lenses separated at a fixed dist- 
ance and by accommodating the eyeballs, enabling one to get 
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3-dimensional (3D) images. There have been many attempts 
to introduce 3D photography in the past and earlier attempts 
were done by first taking two separate images at a slightly dif- 
ferent angle and placing the two frames on dedicated mount 
to get stereographic images for the viewer. However, compli- 
cated set-up and reliance on 3D glasses meant that they were 
too complex or cumbersome for widespread use. 

Recently, 3D image began to gain attention again. Horse- 
man/Rollei unveiled 3D stereo film camera in 2006, but there 
was limitation of still having to use a stereoscopic viewer to 
view 3D images. In 2009, Fujifilm Holdings Corporation 
(Tokyo, Japan) has launched the first digital 3D camera with 
more advanced technology enabling application of 3D sheet 
to the liquid crystal display monitor obviating the need of 
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using stereoscopic viewer, thus allowing the viewers to view 
the 3D image directly on the screen. In addition, filmmakers 
are competitively producing 3D movies and enticing the 
public to the world of 3D technology. These recent changes 
seem to herald the commencement of a new era of 3D tech- 
nology. The 3D imaging technique is expanding its realm, 
especially into the field of medicine; in fact, it has already 
been used in the fields such as general surgery, neurosurgery, 
or urology, guided by laparoscopes and robots. 1 ' 2 

Since its introduction, laparoscopic surgery has revolution- 
ized the field of surgery in many aspects such as reduced post- 
operative pain, shorter recovery time and hospital stay, earlier 
return to everyday living, fewer postoperative complications, 
etc. However, indirect viewing on a 2-dimensional (2D) sc- 
reen rendered accurate manipulation of the instruments diffi- 
cult. 3 Due to poor depth perception, relative position of the 
anatomic structures had to be perceived mainly by touching 
the anatomic parts and previous experiences of the surgeon. 
Thus, it has been shown that compared with surgery under 
direct normal vision, 2D endoscopic vision impaired surgical 
performance by 35% to 100%. 3 ' 4 In order to overcome this 
problem, stereoscopic imaging was applied to laparoscopy. 
Although the usefulness of the stereo viewing varied among 
studies, 5 it was proved to be beneficial for novice surgeons 
and in carrying out complex tasks. 6 

In the field of gastroenterology, 3D imaging technique has 
gained some attention in part for virtual colonoscopy. How- 
ever, as the name implies, it is "virtual" and thus there exists 
discrepancies between direct and virtual visualization; virtual 
colonoscopy cannot accurately visualize polyps of less than 1 
cm in size, cannot tell subtle differences in lesions by texture 
and color, and may misinterpret retained feces as polyps. 7 To 
diagnose and discern such lesions accurately, applying and 
obtaining 3D images of the gastrointestinal tract with endo- 
scopes seems to be an intriguing and a plausible idea. For the 
moment, diagnosis and treatment of mucosal lesion would 
seem sufficient with current gastrointestinal endoscopes. Ne- 
vertheless, with the advent of natural orifice transluminal en- 
doscopic surgery (NOTES), endoscopists are now confront- 
ed with the same limitations as laparoscopic surgeons had to 
face, i.e., depth perception. It is very likely that conventional 
endoscopes will continue to be used for NOTES procedures 
performed via transesophageal, transgastric, or transcolonoic 
route. Therefore, as a first step in assessing the feasibility of 
applying this attractive 3D concept to conventional gastroin- 
testinal endoscopy, we conducted an experiment to evaluate 
whether images obtained with conventional gastrointestinal 
endoscopes could be used to acquire quantitative 3D infor- 
mation. 



MATERIALS AND METHODS 
Materials 

To obtain data for 3D image analysis, two endoscopes (GIF- 
H260, Olympus, Tokyo, Japan) were placed side-to-side, fixed 
at a predetermined distance (12 mm) from each other, and en- 
doscopic stereo-images of an object were obtained. The object 
observed with endoscopes was a Borrmann type I tumor mo- 
del made of clay that was 40 mm high and 43 mm in diameter. 

Image preprocessing 

In the endoscopes, fish-eye lens are utilized to get a wider fi- 
eld of view of the gastrointestinal lumen. However, the use of 
fish-eye lens causes barrel distortion producing images quite 
different from the true image as shown in Fig. 1. The image 
becomes more distorted as the object is observed at a closer 
distance. In addition, when a 3D image is projected on a 2D 
plane, another distortion called perspective distortion occurs 
and the resulting image without correction of the distortions 
would not be fit for further analysis and be prone to a wide 
range of errors. Therefore, before processing the acquired 
image for further analysis, the endoscopes were calibrated by 
correcting the barrel distortion and perspective distortion (Fig. 
2). Endoscope calibration was carried out by using MATLAB 
Toolbox (MathWorks, Natick, MA, USA) which enabled us 
to acquire the focal length (/) of the fish- eye lens in the endo- 
scope. Afterwards, the obtained image was converted to gray- 
level image and the characteristics of the image was obtained 
by edge detection. Finally, data on 3D parameters were mea- 
sured by using epipolar geometry, two view geometry, and a 
pinhole camera model. 

Epipolar geometry, two view geometry, and 
a pinhole camera model 

There are few terminologies that are worth mentioning. St- 
ereo-image is an image captured by two cameras. Matching is 
the process of finding the corresponding position of the point 
on the left image with that of the right image. Disparity indica- 
tes relative position of a point in an image (Fig. 3). 

After stereo-images have been taken from different view- 
points and the images have been preprocessed, epipolar geo- 
metry then describes the relation between the two resulting 
views. In brief, epipolar geometry is a simple but essential 
geometric concept which explains that when multitude of po- 
ints connecting M, M h M 2 , etc. are observed from C L , it is ob- 
served as a dot on its image plane as m L , but when these points 
are observed from C R , it is observed as a line on its image 
plane as e R -m E (Fig. 4). It is then assumed that the cameras are 
pinhole cameras (C L and C R ), which has image planes in front 
of the focal point-unlike real cameras which have image pl- 
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Fig. 1. An example of distorted image by an endoscope. Fish-eye lens utilized in endoscope causes barrel distortion of the original image 
(A), rendering distorted images (B, C, D). 






Original image Barrel distortion corrected Perspective distortion corrected 

Fig. 2. Correction of barrel distortion and perspective distortion. To be able to use images obtained by endoscopes for further analysis, the 
original image should be preprocessed by correcting the barrel distortion and perspective distortion. 



anes behind the focal point-and that all dots lie on the same 
plane called epipolar plane which connects M, C L , and C R . 
This is the fundamental concept in reprocessing and calcul- 
ating the parameters of 3D images. 

Now, we take the epipolar plane to calculate the distance 
between the camera and the object by triangulation using two 
view geometry (Fig. 5). In this figure we can see that the im- 



age plane has been placed behind the focal points as is with 
real cameras. The /has been previously obtained by calibrat- 
ing the camera, which in our case is the endoscope. The dist- 
ance between the two endoscopes (dn) has been predeter- 
mined. When the object "M" is seen from two endoscopes 
C L and C R , two corresponding images m L and m R can be ob- 
tained on the each image plane of C L and C R . Since ra L and m R 
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Fig. 3. An example of stereo-images. Stereo-image is a pair of 
images captured by two cameras. Matching is the process of 
finding the corresponding position of the point on the left image 
with that of the right image. Disparity is relative position of a point 
in an image. 
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Fig. 4. Epipolar geometry. When the points connecting M, M<\, M 2 , 
etc. are observed from G_, it is observed as a dot on its image 
plane as mi, but when these points are observed from Cr, it is 
observed as a line on its image plane as ©r-at7r. 

are images of the same object, it can be matched, but due to 
the difference in the relative location of the object that is pro- 
jected onto corresponding planes, there is disparity (v 2 -Vi). 
Equation on disparity is as follows: 



Disparity=v 2 -Vi= 



L 



(Equation 1) 
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Fig. 5. Two view geometry of cameras. In this model, c/12 is the 
distance between two cameras, f the focal length, L the distance 
between the cameras and the object, and v<\-v 2 the disparity. 
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If we substitute all the known and obtained variables, we 



Fig. 6. Pinhole camera model. In this model, f is the focal length, 
C the camera centre, c the image centre, h the image point of H, 
and L the distance between the camera center and the object. 

can calculate the distance between the cameras and the ob- 
ject (L). 

After "L" has been determined, we can calculate the height 
of the object (H). This can be done by using a pinhole camera 
model in which/is focal length, h is the height of the project- 
ed image, and L is the distance obtained from the previous 
equation (Fig. 6). When we substitute these data into the eq- 
uation below, 

/: h=L : H (Equation 2) 

we can get the calculated height of the actual object. 

RESULTS 

Stereo-images were obtained at three different distances 
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Fig. 7. The stereo-image of Borrmann type I tumor model. The matching point was (left, 374, 241) and (right, 265, 231), and the resulting 
disparity was 106 pixels. 

Table 1. Results of the Calculated Distance (/_) and the Measurement Error between the Endoscope and the Borrmann Type I Tumor Model 





f, pixel dn, mm 


V2-V1, pixel 


L, mm 


Actual distance, mm 


Error, % 


Stereo-image 1 


258.49 12 


106 


29.26 


30 


2.46 


Stereo-image 2 


297.19 12 




89 


40.07 


40 


0.18 


Stereo-image 3 


316.30 12 




75 


50.61 


50 


1.22 


/, focal length; dn, distance between two camera; v 2 -Vi, disparity; L, distance between the cameras 


and the object. 




Table 2. Results of the Calculated Height and Width, and the Measurement Error of the Borrmann Type 1 Tumor Model 








hy pixel 




H y mm 


Actual size, mm 


Error, % 


Stereo-image 1 


Height 


337.5 




38.21 


40 


4.48 




Width 


368.5 




41.72 


43 


2.98 


Stereo-image 2 


Height 


280.5 




37.82 


40 


5.45 




Width 


308.5 




41.60 


43 


3.27 


Stereo-image 3 


Height 


232.5 




37.20 


40 


7.00 




Width 


252.5 




40.40 


43 


6.05 


/z, image point of H; H, calculated image size (height and weight). 



from the Borrmann type I tumor model: 30, 40, and 50 mm. 
The /obtained by calibrating the endoscope using MATLAB 
Toolbox at 30 mm from the Borrmann type I tumor model 
was 258.49 pixels. Two endoscopes were fixed at a predeter- 
mined distance, 12 mm (dn). The corresponding points of the 
Borrmann type I tumor model obtained by each endoscopes 
(374, 241) and (265, 231), were matched, and the disparity 
V2-V1 was calculated as 106 pixels (Fig. 7). When all these val- 
ues were substituted into the equation 1, the length between 
the camera and the object (L) was 29.26 mm (Table 1). The 
height and width of the object projected onto the image (h) 
was then substituted into the proportional equation 2, and 
the result was 38.21 mm and 41.72 mm, respectively (Table 2). 
The values obtained at 40 mm and 50 mm are also shown in 
Tables 1,2. 

The measurement error of the distance between the endo- 
scope and the current Borrmann type I tumor model ranged 



from 0.18% to 2.46%, and that of the height and width rang- 
ed from 2.98% to 7.00%. 

DISCUSSION 

Through this experiment, we could see that acquiring quan- 
titative 3D information by applying epipolar geometry, two 
view geometry, and a pinhole camera model to calculate the 
size of an object was feasible with conventional endoscopes. 
The pinhole camera model accompanied by correction of bar- 
rel and perspective distortions, as had been done in our study, 
is known to be a fair approximation for most of the narrow- 
angle and wide-angle lenses. 8,9 However, since there is an in- 
herent distortion of a fish-eye lens that makes it difficult to be 
considered a mere deviation from a pinhole camera model, 10 
errors have occurred in our measurements. Therefore, empl- 
oying a generic camera model proposed by Kannala and Br- 
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andt, 11 which is based on viewing a planar calibration pattern 
instead of perspective projection used by the pinhole camera 
model, could have conferred greater precision in obtaining 
parameters. For simplicity of calculation, however, we have ap- 
plied the pinhole camera model in our study to calculate the 
H and have produced a fair approximation. Nevertheless, ap- 
plying the generic camera model should be considered in re- 
constructing 3D images in the future studies. 12 In addition, the 
errors in measurement might have been due to the fact that 
the feature points of an object could not be accurately extract- 
ed. This seems to have risen because the tumor model we 
made did not have definite edges. If the model used for mea- 
surement had been a cube or an object with definite edges, fe- 
ature point extracting and matching would have been facili- 
tated. However, we intended to simulate an actual tumor even 
though this had been a source of errors. One way to overcome 
feature point extracting and matching problems with an ac- 
tual tumor could be by applying the method called circular 
hough transform. In a preliminary study by our group, the cir- 
cular hough transform was shown to be able to extract feature 
points from 2D tumor stereo-images to recover information 
with good accuracy. 

Light source seems to be another problem in obtaining im- 
ages. In the initial phase of this experiment, we tried to obtain 
images by using two different models of endoscopes, GIF- 
H260 (Olympus) which had stronger illuminating power and 
GIF-Q260 (Olympus) which had relatively weaker one. How- 
ever, while the image taken with GIF-H260 took on a true col- 
or, the image taken with GIF-Q260 took on a bluish hue, be- 
ing dominated by the stronger light. The differences in illu- 
minating power of the two endoscopes lead to skewed color 



images. In addition, the size of the projected image was dif- 
ferent with each endoscope. Since it should be like capturing 
images individually and twice, once with the right eye and 
another with left eye, perfect synchronization of the right and 
left images is mandatory and is of the utmost importance. 
When different types of endoscopes were used, this could not 
be achieved due to the lack of concordance of range and angle 
of the view. Therefore, we decided to use two endoscopes of 
the same model and had equal illuminating power. Never- 
theless, we were faced with other problems: light interference 
and mixture or addition of lights. Although two endoscopes 
of the same model were used, they were exact twins but two 
separate endoscopes with discordance of brightness, color, 
and image quality. The pictures taken during our preliminary 
experiment demonstrate artifacts resembling a comets tails, 
as if the tumor model in the picture was moving towards each 
other and leaving behind its trace (Fig. 8). This is due to light 
interference or discordance of brightness. In addition, the use 
of two light sources of equal illuminating power resulted in 
the mixture or addition of lights rendering only a white tone. 
This is due to discordance of color. This could be overcome 
by making an endoscope with two lenses that uses a single 
light source which in current endoscopes would be two illu- 
mination lights that are synchronized. Recently, even more 
ideal and unique 3D system has been developed by Wasol 
(Seoul, Korea). Unlike all the other 3D systems that are based 
on two camera system, Wasol's 3D endoscope applies only 
one camera. Changing the path of light using the difference 
in density between objects enables one to obtain stereo im- 
ages. 13 This novel 3D system would essentially eliminate the 
discordance problems related to zooming, focusing, and align - 




Fig. 8. The stereo-image of preliminary Borrmann type I tumor experimental model. When two endoscopes of the same model and equal il- 
luminating power were used simultaneously to obtain stereo-images, light interference occurred, leaving behind artifacts resembling a comet's 
tails. 
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ing when two lenses are used. 

In addition to the problem with light source, another pro- 
blem to overcome was that the 3D parameters acquired from 
our study were calculations from 2D still images and were not 
real time based. Although we have demonstrated the possi- 
bility of extracting 3D information from 2D images using 
conventional endoscopes, applying the endoscope system 
presented by Nakatani et al. 14 which has four laser beam so- 
urces attached to the endoscope could also be considered in 
constructing future 3D endoscopes to facilitate the acquisi- 
tion of real time measurements of the size and position of an 
object. 

In our study, two separate endoscopes were used alternately 
to recover a set of image. Hasegawa and Sato 15 devised a 3D 
endoscope to realize high-speed 3D measurement which also 
incorporated laser scanning. This endoscope was a rigid en- 
doscope that has two lenses in the shaft allowing binocular 
3D viewing and was subject to some measurement errors de- 
pending on the movement of the object being observed. Al- 
though it might be necessary at first to use a binocular viewer 
or other special viewer to observe a lesion in 3D, rapidly ad- 
vancing technology will enable adopting a 3D display that 
will allow endoscopists to view the lesion right on the screen. 
In addition, the problem of increased endoscope diameter 
due to the incorporation of two lenses might be overcome by 
adopting the above mentioned Wasols 3D system. 

We still have a long way to go before 3D technology is fully 
integrated into endoscopy. However, with further advances 
in technology, it will lessen the burden of surgeons during 
operation and help the gastrointestinal endoscopists during 
such procedures as endoscopic submucosal dissection and 
NOTES procedures. They will not only be able to observe a 
lesion in real time 3D view but measure depth, exact size, 
volume, etc. to aid their judgment during observation and 
procedure. Subtle mucosal change might also be discerned 
with greater accuracy. Further progress will enable us to ch- 
aracterize the lesions leading to formulation of a new classifi- 
cation of polyps and cancers, new evaluation index of resp- 
onse to chemotherapy, and novel criteria for assessing ulcer 
healing response by measuring the 3D structure of an ulcer. 16 
There is no doubt that problems related to depth perception 
during NOTES procedures, as has occurred with convention- 
al laparoscopy, could be overcome. Applying 3D technology to 
endoscopes will add new dimension to endoscopy. Further- 
more, easy to use 3D endoscopes will prove to be as impor- 
tant a step in imaging as the introduction of digital endo- 
scope a decade ago. Although there are many improvements 
to be made before meeting our expectations, the doors for 



future development of 3D endoscopy are open to revolution- 
ize and make great contribution in the progress of a new era 
of endoscopy. All endoscopists should keep their eyes open 
since 3D endoscopy might prove to be something more than 
meets the eye. 
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